Skip to content

feat: run gateway tool loop natively on /v1/messages#131

Merged
maralbahari merged 3 commits into
vllm-project:mainfrom
ashwing:feat/messages-gateway-tool-loop
Jul 21, 2026
Merged

feat: run gateway tool loop natively on /v1/messages#131
maralbahari merged 3 commits into
vllm-project:mainfrom
ashwing:feat/messages-gateway-tool-loop

Conversation

@ashwing

@ashwing ashwing commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Part of #113; implements #115 (Stage 2).

What

Run the gateway tool loop natively on /v1/messages: when a request declares a gateway-owned tool, the gateway executes it server-side, hides it, and returns only the final assistant message. No gateway tool → unchanged transparent proxy.

This also makes Claude Code's web search run on the gateway — the Codex experience, for Claude Code.

How

The loop is native to Anthropic Messages — it talks vLLM /v1/messages upstream and forwards the client's request untouched, so every Anthropic field (tool_choice, stop_sequences, thinking, …) is preserved. It reuses only the protocol-neutral tool layer (ToolRegistry::dispatch) via a small seam; it never touches RequestPayload/ResponsePayload.

  • types/messages/tool_seamtool_useFunctionToolCall, ToolOutputtool_result, gateway/client classification.
  • executor::messages_loop — non-streaming loop (hide-the-call, feed tool_result back, round cap).
  • executor::messages_stream — streaming loop + accumulator presenting one logical message across rounds (single lifecycle, contiguous block indices, gateway tool_use suppressed).
  • handler — routes gateway-tool requests to the loop; Anthropic error envelope on failure.

Claude Code web search

Claude Code declares web search as a client tool named WebSearch (not a typed server tool like Codex's web_search), so it is client-owned by default. Operators opt in with MESSAGES_GATEWAY_TOOL_ALIASES="WebSearch=web_search" (empty by default) to have the gateway execute it — the ownership doctrine's "client function is client-owned unless configured as gateway-owned" (codex-integration.md). Dispatch canonicalizes the name to the web_search executor and adapts the arguments (allowed_domainsinclude_domains, blocked_domainsexclude_domains). See the README's Claude Code section and docs/design/messages-gateway-tool-classification.md.

Tests

  • cargo fmt --check / clippy --workspace --all-targets -D warnings / cargo test --workspace — clean.
  • Unit: tool seam (classification, alias parsing, name canonicalization, arg adapter), streaming accumulator.
  • Acceptance: replay live-recorded /v1/messages cassettes — single-round, parallel, sequential multi-round, no-tool-call, streaming.
  • Edges: tool failure/timeout → error tool_result, upstream error → Anthropic envelope, round cap, malformed args, mixed client+gateway tools, proxy fallthrough.
  • Live (G6e, Qwen3 + real You.com): curl scenarios; real Codex CLI (typed web_search) and real Claude Code CLI (aliased WebSearch) both drove the loop server-side with the call hidden — routed messages_loop, live search backend hit, single streaming lifecycle, no raw-Responses leakage.

Known limitations (intentional)

  • pause_turn surfaced, not continued — only stop_reason: tool_use drives another round; other stops return to the client.
  • MCP over Messages out of scope — its wire shape isn't expressible in the Anthropic tool declaration yet.
  • WebSearch domain filters — the executor treats include/exclude domains as mutually exclusive, so a call setting both allowed_domains and blocked_domains returns a graceful error result.

Follow-up: consolidation with the Responses loop

This adds a second gateway-tool loop alongside the Responses one; the duplication is deliberate (different wire shapes; the typed core shouldn't be reshaped ahead of the Layering ADR). Tool execution is already shared. Planned follow-ups (post-ADR): a GatewayTurn trait + one generic loop, a shared stream-normalizer contract, hoisted shared limits, and structural typed-tool + per-request opt-in classification on the Messages side (extends the same GatewayToolMap seam).

@ashwing
ashwing force-pushed the feat/messages-gateway-tool-loop branch from 76d4433 to f7a7759 Compare July 16, 2026 20:50
@ashwing ashwing changed the title feat: run gateway tool loop on /v1/messages (non-streaming) feat: run gateway tool loop on /v1/messages Jul 16, 2026
@ashwing
ashwing force-pushed the feat/messages-gateway-tool-loop branch 6 times, most recently from 69fffd3 to 229070d Compare July 17, 2026 03:18
@ashwing
ashwing marked this pull request as ready for review July 17, 2026 03:32
@ashwing
ashwing marked this pull request as draft July 17, 2026 05:14
@ashwing
ashwing force-pushed the feat/messages-gateway-tool-loop branch from 229070d to 6116f55 Compare July 17, 2026 06:20
@ashwing ashwing changed the title feat: run gateway tool loop on /v1/messages feat: run gateway tool loop natively on /v1/messages Jul 17, 2026
When an Anthropic Messages request declares a gateway-owned tool (web_search),
run the server-side gateway tool loop natively against vLLM /v1/messages —
execute the tool, hide it from the client, and surface only the final
assistant message. Requests without a gateway-owned tool keep the transparent
proxy (vllm-project#99).

The loop talks Anthropic Messages end to end (no RequestPayload/ResponsePayload
detour), so the client's request is forwarded to vLLM untouched and every
Anthropic field is preserved. It reuses only the protocol-neutral tool layer
(ToolRegistry::dispatch, name-based classification, per-call timeout,
concurrent execution) via a small tool seam.

- types/messages: Anthropic wire types + tool seam (tool_use<->FunctionToolCall,
  ToolOutput->tool_result, gateway/client classification).
- executor::messages_loop: non-streaming loop (hide-the-call, feed tool_result
  back, round cap; client-owned tool_use returns to the client).
- executor::messages_stream: streaming loop + MessagesStreamAccumulator that
  presents one logical message across rounds (single lifecycle, contiguous
  block indices, gateway tool_use suppressed, client-owned tool_use terminal) —
  the Anthropic-native analogue of the Responses GatewayAccumulator (vllm-project#119).
- handler: routes gateway-tool requests to the loop, forwards the client's
  x-api-key, returns the Anthropic error envelope on failure.

Tests: tool-seam + accumulator unit tests; non-streaming and streaming
acceptance tests replaying live-recorded /v1/messages cassettes (single-round,
parallel, sequential multi-round, no-tool-call); edge coverage for tool
failure, upstream error, round cap, malformed input, and mixed client+gateway
tools; handler routing tests. Validated live on G6e (Qwen3 + real You.com) via
curl and a real upstream Claude Code CLI session.

Implements vllm-project#115.

Signed-off-by: Ashwin Giridharan <girida@amazon.com>
@ashwing
ashwing force-pushed the feat/messages-gateway-tool-loop branch from 6116f55 to 28a989c Compare July 17, 2026 07:36
@ashwing
ashwing marked this pull request as ready for review July 17, 2026 07:48

@maralbahari maralbahari left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashwing thanks for the PR. I added some inline comments

Comment thread crates/agentic-server-core/src/executor/messages_stream.rs Outdated
Comment thread crates/agentic-server-core/src/executor/messages_stream.rs
Comment thread crates/agentic-server-core/src/executor/messages_loop.rs Outdated
Comment thread crates/agentic-server-core/src/executor/messages_stream.rs Outdated
Comment thread crates/agentic-server-core/src/executor/messages_loop.rs Outdated
Comment thread crates/agentic-server-core/src/executor/messages_stream.rs
Address review on the Messages-native gateway tool loop:

- Feed the model's full assistant turn (thinking/text/signature +
  tool_use, order preserved) into the next round's history in both the
  streaming and non-streaming loops, instead of reconstructing it from
  only the gateway tool_use blocks. Dropping the preceding blocks lost
  conversation state and could invalidate extended-thinking follow-ups.
- Reject malformed/incomplete tool input (streaming partial_json or a
  non-object input) with an is_error tool_result rather than coercing to
  {} and dispatching the tool with arguments the model never supplied.
- Apply a consistent mixed-call policy: when an assistant turn contains
  both a gateway-owned and a client-owned tool_use, hide the gateway
  call, surface the client call, and stop the loop -- in both paths.
- Reset final_message_delta in begin_round() so a clean-EOF round cannot
  re-emit the previous round's stop_reason: tool_use.

Adds repro tests covering each case in both loops.

Signed-off-by: Ashwin Giridharan <girida@amazon.com>
/// Gateway-tool execution failures do **not** error — they become error
/// `tool_result`s fed back to the model.
pub async fn run_messages_loop(
mut request: Value,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why instead of raw json value not using MessagesRequest that you have in types/messages.request.rs ?

@maralbahari maralbahari Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashwing if the raw lossless json is really required maybe consider passing a per-request context containing both the typed MessagesRequest for safe field access and the raw JSON Value for lossless upstream forwarding? or you can follow the pattern we use for RequestPayload for Responses API where it implements to_upstream function. so wouldnt need to maintain a context where it holds lossless for upstream and typed version for gateway.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair question. The difference from RequestPayload::to_upstream_request is that that path transforms — it flattens Codex namespace tools, normalizes every tool type to Vec<FunctionTool>, resolves tool_choice. Typed→typed is the actual work there. This loop is a pass-through: it forwards the client's request to vLLM /v1/messages verbatim and only reads tools + rewrites messages/stream, so there's no transform step a typed request would carry.

One concrete catch with going fully typed here: ContentBlock has #[serde(other)] Unknown, so deserializing→reserializing through MessagesRequest would silently drop any block type we don't model (redacted_thinking, future block types, provider extensions). Raw Value forwards those untouched, which matters for a proxy.

That said — you're right that a per-request context (typed MessagesRequest for safe field access + raw Value for lossless forwarding) is the cleaner long-term shape, and it lines up with the loop-consolidation follow-up already noted in the PR description. I'd rather not reshape the request plumbing inside this PR (it'd touch the routing path and widen the diff), so I'll file it as a follow-up and link it here. Sound good?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashwing thanks ashwin, you can create Issue for this for the follow up. approving this PR.

ashwing added a commit to ashwing/agentic-api that referenced this pull request Jul 20, 2026
Records the plan to classify gateway-owned tools by shape (not name) on
the Messages side so Claude Code's web search and MCP can run through the
gateway, mirroring the Responses side. Grounded in live testing: Codex
declares web search structurally and already routes through the loop;
Claude Code declares a client-owned WebSearch function that proxies. A
name alias was rejected as a client-tool hijack. Follow-up to vllm-project#131.

Signed-off-by: Ashwin Giridharan <girida@amazon.com>
@ashwing
ashwing force-pushed the feat/messages-gateway-tool-loop branch 3 times, most recently from 58fa6bb to 83ca75b Compare July 20, 2026 21:40
…e Code)

Claude Code declares its built-in web search as a client function named
`WebSearch`, so it fell through the name-based `web_search` classifier and
was proxied — Claude Code's web search never ran server-side, unlike
Codex, whose typed `web_search` tool routes through the Responses loop.

Add an operator-configured client-tool -> gateway-executor alias map
(`MESSAGES_GATEWAY_TOOL_ALIASES`, e.g. `WebSearch=web_search`), empty by
default. This is the ownership doctrine's "client function is client-owned
unless configured as gateway-owned" clause (docs/design/codex-integration.md),
applied structurally rather than by a hardcoded name. When configured:

- routing, registry build, and both loops classify the aliased tool as
  gateway-owned via GatewayToolMap;
- dispatch canonicalizes the call name to the `web_search` executor key AND
  adapts the arguments (allowed_domains -> include_domains,
  blocked_domains -> exclude_domains) so Claude Code's WebSearch schema
  reaches the executor correctly;
- the raw request still forwards to vLLM verbatim, so the model sees the
  client's own tool contract; the gateway executes and hides the call.

Adds the follow-up design doc and unit tests for classification, env
parsing, name canonicalization, and the argument adapter.

Signed-off-by: Ashwin Giridharan <girida@amazon.com>
@maralbahari
maralbahari merged commit 7783068 into vllm-project:main Jul 21, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants